SHRADDHA SWAROOP

Customer requirements

The second step in the process is to understand the context in which the tool will be used in. The requirements were set by the customer and written together by both Kathryn and Shraddha.

1. Use case: Researchers and tech leads involved in a HDV flight test or simulation said a lack in transparent communication about a preceding step being accomplished could delay them in the step he/she is responsible for. This lag caused the system to log out and restarting the system caused lateness in accomplishing their step in the process.
• • Create a communication tool that allows researchers to the status of what step is being conducted in the live flight test or simulation.
• • This tool shall cue researchers to engage specific tasks as appropriate for their role.
• • This tool shall also tell the researcher how preceding steps affects their own tasks.
• • This tool shall not distract or interfere with research activities.

Goals of evaluation

Determine how well the existing iteration of the prototype provides readily-available, easily-accessible and important information to current and researchers (task performance). Identify usability issues that hinder access to relevant information or cause frustration in those using this tool (user satisfaction).

Method being used

Efficiency, effectiveness and usability of this product will be evaluated using a moderated formative usability test which is appropriate for iterative design, limited participants and quick turnaround time.

Metrics being used

• • Time on task
• • Task success rate
• • Self-reported task difficulty
• • System usability scale (SUS) for websites

RESULTS

Participant demographics are presented first, followed by summary data, and a detailed analysis of the three least problematic tasks and the three most problematic tasks.

Participant Demographics

• • Gender: Female (2), Male (4)
• • Ages: 20-30 (4); 31-40 (2)
• • Roles: Tech lead (1); Researchers (4); Developer (1)
• • Experience: 0 to 5 years (4); 6 to 10 years (1); 11 to 15 years (1); 16+ years
• • Base: NASA-Ames (4); NASA-Langley (2)
• • HDV Flight or Simulation experience: 0 to 5 years (4); 6 to 10 years (0); 11+ years (1); Other (0)
• • Team situation awareness: (1=weak and 5 is strong): 1 (0); 2(0); 3(3); 4(2); 5(1);

Quantitative results

An overview of the data for each metric utilized to determine usability of the Website is as follows:

System Usability Scale

The SUS is a self-report measure that asks participants 10 questions relating to their perceptions of the communication tool prototype. The odd-numbered questions are positively-worded and the even-numbered questions are negatively-worded. The SUS is scored as follows:

• • Odd-numbered questions: subtract 1 from the participant’s rating;
• • Even-numbered questions: subtract the participant’s rating from 5;
• • Add up the new scores then divide by 2.5;
• • Scores range from 0 – 100;
• • Scores < 50 = unacceptable usability;
• • Scores 50 – 70 = marginal usability;
• • Scores > 70 = acceptable usability.

The SUS score for each of our participants appears below. The green bar represents the minimum score required to obtain acceptable usability. Here, the average SUS score was almost 84, which is well above acceptable usability.

Completion time, accuracy rate and self-reported task difficulty

Data are summarized by task in the table below

Time on Task Summary Data

To find the central point of the time on task data, we used the geometric mean, as it is the most appropriate when analyzing centrality of highly skewed and variable data such as time-related data. The optimal task time, range, and standard deviation are also provided in the chart below.

Task key

• Task #1: Log in as a GCSO researcher and obtain security clearance
• #2: Find the upcoming task for Range Safety Officer
• #3: Find upcoming task for yourself, the GCSO researcher
• #4: Check off the next test for yourself, the GCSO researcher, to let the team know that step is complete.

The two tasks with the highest geomean completion times were as follows:

• • Task 2 (Find the upcoming task for Range Safety Officer): 58.28 seconds;
• • Task 3 (Find upcoming task for yourself the GCSO researcher): 84.68 seconds.

The above summary results allowed our team to easily identify the single most problematic task and the two least problematic tasks, each of which are discussed in detail below. Because our test tasks mainly related to finding information on the prototype tool, we identified Time on Task, and Success Rate as the most important measures on which to focus our detailed evaluation.Listed below are the three least problematic tasks. As these tasks had relatively high success rates, relatively low task times, and low difficulty ratings, the remainder of this report will focus on the one task that we deemed to be the most problematic.

Three Least Problematic Tasks

To find the central point of the time on task data, we used the geometric mean, as it is the most appropriate when analyzing centrality of highly skewed and variable data such as time-related data. The optimal task time, range, and standard deviation are also provided in the chart below.

Task	Success rate	Geomean time on task (seconds)	Optimal time* (seconds)	Mean difficulty rate (out of 5)
#1: Log in as a GCSO researcher and obtain clearance	6/6	22.20	3.97	4.83
#2: Find the upcoming task for Range Safety Officer	6/6	58.28	5.90	4.33
#4: Check off the next test for yourself, the GCSO researcher, to let the team know that step is complete.	6/6	19.61	5.50	4.00

*Note: Optimal time was determined by team members who were familiar with the fastest way to accomplish the task. Participants were unfamiliar with where to find the information and were also thinking out loud as they were completing the tasks. As such, the discrepancies between the optimal time and participants’ actual times were expected.

Single Most Problematic Task

As stated above, our team focused on Time on Task and Success Rate as the most important metrics on which to focus our detailed evaluation of the single most problematic task which was Task #3.

Task	Success rate	Geomean time on task (seconds)	Optimal time* (seconds)	Mean difficulty rate (out of 5)
#3: Find upcoming task for yourself, the GCSO researcher.	4/6	84.68	4.48	4.25

Task # 3: Find upcoming task for yourself, the GCSO researcher

• Success rate: 4/6; (One participant could not find this information immediately and eventually made it to the My Tasks tab to complete the activity but exceeded the 2 minute time limit; the other participant did find the information but also exceeded the time limit; the other four found this information on the Home Page instead of the My Tasks page.
• Geomean task time: 84.68 seconds
• Optimal time: 4.48 seconds
• Difficulty score: 4.25/5.0



Half of the subjects saw the upcoming tasks for the GCSO researcher on the Home page and either didn’t see the My Tasks button or were waiting for it to light up to go there. One person did eventually get there and the other did not. The participant did not get there but did go there for Task 4 when was trying to check off the task as completed. The information about the next task can be found on the Home Page but it is static. The only way to interact with it is to go to the My Tasks page by clicking the My Tasks link in the navigation.




Four participants were able to complete this, but the majority expressed frustration at having to go to a whole new page to see different tasks based on different roles:

• • “So that's also something that I guess wasn't maybe obvious to me. ”
• • “I would think that the interface would make it more clear which ones are mine versus theirs, right? Cause that could be confusing if there's more than one.”
• • “OK, so I would want to check this (on the Home Page). I would try to click here. But that doesn't work, so I need to go over here to My Tasks.”